Papers with neural representations
Privacy-preserving Neural Representations of Text (D18-1)
Copied to clipboard
| Challenge: | a specific type of attack is used to characterize the privacy of neural representations for NLP tasks, in the context of privacy protection. |
| Approach: | They propose several defense methods based on modified training objectives and characterize the tradeoff between privacy and the utility of neural representations. |
| Outcome: | The proposed defenses improve the privacy of neural representations and characterize the tradeoff between privacy and utility of representations. |
Knowledge-Enriched Natural Language Generation (2021.emnlp-tutorials)
Copied to clipboard
| Challenge: | Knowledge-enriched text generation poses unique challenges in modeling and learning . a roadmap will outline the state-of-the-art methods to tackle these challenges . |
| Approach: | They propose a roadmap to tackle the challenges of knowledge-enriched text generation . they will dive deep into various technical components to illustrate how to represent knowledge . |
| Outcome: | This tutorial outlines the state-of-the-art methods to tackle the problem . it aims to show how to represent knowledge, feed knowledge into a generation model, evaluate results . |
Transformers for Tabular Data Representation: A Survey of Models and Applications (2023.tacl-1)
Copied to clipboard
| Challenge: | Recent research efforts extend LMs by developing neural representations for structured data. |
| Approach: | They propose to extend transformer-based language models to tabular data by analyzing inputs, model training, and supported downstream tasks. |
| Outcome: | The proposed models are compared against existing models and are based on a traditional pipeline. |
Neural Token Representations and Negation and Speculation Scope Detection in Biomedical and General Domain Text (D19-62)
Copied to clipboard
| Challenge: | Existing evidence for improved performance on natural language tasks is unclear to what degree the learned token representations capture and encode highlevel morphological/syntactic knowledge about the usage of a given token in a sentence. |
| Approach: | They propose to use context-aware token representations to capture morphological/syntactic knowledge about the usage of a given word/token in a sentence. |
| Outcome: | The proposed representations capture and encode high-level morphological/syntactic knowledge about the usage of a given token in a sentence. |
Erasure of Unaligned Attributes from Neural Representations (2023.tacl-1)
Copied to clipboard
| Challenge: | Developing a methodology for adjusting neural representations to preserve user privacy and avoid encoding bias in them is an active area of research in recent years. |
| Approach: | They propose an algorithm which erases information from neural representations when the information to be erased is implicit rather than directly aligned to each input example. |
| Outcome: | The proposed algorithm erases information from neural representations when the information to be erased is implicit rather than directly aligned to each input example. |
A Differentiable Integer Linear Programming Solver for Explanation-Based Natural Language Inference (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing ILP frameworks are non-differentiable and cannot be integrated as part of a broader deep learning architecture. |
| Approach: | They propose a neuro-symbolic architecture for explanation-based NLI based on DBCS. |
| Outcome: | The proposed approach achieves superior performance when compared to existing solvers and black-box solver. |
CogTaskonomy: Cognitively Inspired Task Taxonomy Is Beneficial to Transfer Learning in NLP (2022.acl-long)
Copied to clipboard
| Challenge: | Existing frameworks for transfer learning across tasks in natural language processing are CRA and CNM . CRA uses a correlation between brain-activity measurement and computational modeling to estimate task similarity with sentence representations. |
| Approach: | They propose a cognitively inspired framework to learn taxonomy for NLP tasks . they use Cognitive Representation Analytics and Cognitive-Neural Mapping . |
| Outcome: | The proposed framework can guide transfer learning across tasks in natural language processing without exhaustive pairwise task transferring. |
Gold Doesn’t Always Glitter: Spectral Removal of Linear and Nonlinear Guarded Attribute Information (2023.eacl-main)
Copied to clipboard
| Challenge: | Spectral Attribute removaL is a method to remove private or guarded information from neural representations. |
| Approach: | They propose a method to remove guarded or private information from neural representations by matrix decomposition. |
| Outcome: | The proposed method retains better main task performance after removing guarded information compared to previous work. |
Conditional probing: measuring usable information beyond a baseline (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for probing representations are limited to predicting part-of-speech . current methods cannot detect when a representation is predictive of just aspects of part- of-seech not explainable by the word identity. |
| Approach: | They propose to condition on the information in a baseline representation to test whether it is predictive of part-of-speech. |
| Outcome: | The proposed method is based on a theory of usable information called V-information and conditions on the information in the baseline. |
Implicit Representations of Meaning in Neural Language Models (2021.acl-long)
Copied to clipboard
| Challenge: | Neural language models (NLMs) encode lexical relations and syntactic structure, but their effectiveness is still unclear. |
| Approach: | They propose to use text as a model to model entities and situations as they evolve throughout a discourse. |
| Outcome: | The proposed models have functional similarities to linguistic models of dynamic semantics and can be learned with only text as training data. |
AudioSAE: Towards Understanding of Audio-Processing Models with Sparse AutoEncoders (2026.eacl-long)
Copied to clipboard
Georgii Aparin, Tasnima Sadekova, Alexey Rukhovich, Assel Yermekova, Laida Kushnareva, Vadim Popov, Kristian Kuznetsov, Irina Piontkovskaya
| Challenge: | Feature steering reduces Whisper’s false speech detections by 70% with negligible WER increase, demonstrating real-world applicability. |
| Approach: | They train Sparse Autoencoders across all encoder layers of Whisper and HuBERT and evaluate their stability, interpretability, and practical utility. |
| Outcome: | The proposed models capture general acoustic and semantic information as well as specific events, including environmental noises and paralinguistic sounds, and disentangle them effectively. |
Reinforced Product Metadata Selection for Helpfulness Assessment of Customer Reviews (D19-1)
Copied to clipboard
| Challenge: | a helpful review is largely concerned with the metadata of its target product . a selector learns from both the key-value product metadata and one of its reviews to take an action . |
| Approach: | They propose a framework that uses product metadata to assess helpfulness of free-text reviews . they use two real-world datasets from amazon.com and Yelp.com to test the framework . |
| Outcome: | The proposed framework can achieve state-of-the-art performance with substantial improvements . it uses two real-world datasets from Amazon.com and Yelp.com . |
Shielded Representations: Protecting Sensitive Attributes Through Iterative Gradient-Based Projection (2023.findings-acl)
Copied to clipboard
| Challenge: | Natural language processing models tend to learn and encode social biases present in the data. |
| Approach: | They propose a method for removing non-linear encoded concepts from neural representations by iteratively training neural classifiers to predict a particular attribute, followed by a projection of the representation on a hypersurface. |
| Outcome: | The proposed method removes non-linear encoded concepts from neural representations. |
Effective Batching for Recurrent Neural Network Grammars (2021.findings-acl)
Copied to clipboard
| Challenge: | RNNGs are hard to scale due to the difficulty of batched training. |
| Approach: | They propose to batch RNNGs where every operation is computed in parallel with tensors across multiple sentences. |
| Outcome: | The proposed RNNG scales faster than existing models and achieves x6 speedup compared to existing C++ DyNet implementation . |
Numeric Magnitude Comparison Effects in Large Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Prior research on the representational capabilities of LLMs evaluates whether they show human-level performance. |
| Approach: | They ask how well popular LLMs capture the magnitudes of numbers from a behavioral lens. |
| Outcome: | The proposed model captures the magnitudes of numbers from a behavioral lens. |
Similar but not the Same: Word Sense Disambiguation Improves Event Detection via Neural Representation Matching (D18-1)
Copied to clipboard
| Challenge: | Event detection (ED) and word sense disambiguation (WSD) are similar tasks, but they require different neural representations. |
| Approach: | They propose a method to transfer the knowledge learned on WSD to ED by matching neural representations learned for the two tasks. |
| Outcome: | The proposed method can be applied to event detection and word sense disambiguation datasets. |
Log-linear Guardedness and its Implications (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for erasing human-interpretable concepts from neural representations that assume linearity are not fully understood. |
| Approach: | They define linear guardedness as the inability of an adversary to predict the concept directly from the representation . they show that a log-linear model can be constructed that indirectly recovers the concept . |
| Outcome: | The proposed model can be constructed that indirectly recovers the erased concept in some cases. |
Interpretable Multimodal Misinformation Detection with Logic Reasoning (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for misinformation detection lack interpretability due to the black-box nature of the neural network. |
| Approach: | They propose a logic-based neural model which integrates interpretable logic clauses to express the reasoning process of the target task. |
| Outcome: | The proposed model can be generalizable across multiple misinformation sources and is based on three public datasets. |
Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection (2020.acl-main)
Copied to clipboard
| Challenge: | Word embeddings, pre-trained language models, and deep learning methods are becoming effective for text classification. |
| Approach: | They propose a method for removing information from neural representations using null-space projection. |
| Outcome: | The proposed method mitigates bias in word embeddings and increases fairness in multi-class classification. |
CompGuessWhat?!: A Multi-task Evaluation Framework for Grounded Language Learning (2020.acl-main)
Copied to clipboard
Alessandro Suglia, Ioannis Konstas, Andrea Vanzo, Emanuele Bastianelli, Desmond Elliott, Stella Frank, Oliver Lemon
| Challenge: | Approaches to Grounded Language Learning focus on a single task-based final performance measure which may not depend on desirable properties of the learned hidden representations. |
| Approach: | They propose an evaluation framework for Grounded Language Learning with Attributes based on three sub-tasks: 1) Goal-oriented evaluation; 2) Object attribute prediction evaluation; and 3) Zero-shot evaluation. |
| Outcome: | The proposed framework evaluates the quality of learned representations with respect to attribute grounding. |
Document-Level Event Role Filler Extraction using Multi-Granularity Contextualized Encoding (2020.acl-main)
Copied to clipboard
| Challenge: | Document-level event extraction requires a view of a larger context to determine which spans of text correspond to event role fillers. |
| Approach: | They propose a multi-granularity reader to dynamically aggregate information captured by neural representations learned at different levels of granularities. |
| Outcome: | The proposed model performs substantially better than previous models on the MUC-4 event extraction dataset. |
An information theoretic view on selecting linguistic probes (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in NLP tasks require a question of how much linguistic knowledge is encoded in neural networks. |
| Approach: | They propose to use diagnostic classifiers to perform supervised classification from internal representations. |
| Outcome: | Empirically, the two proposed criteria lead to results that agree with each other. |